# Small Language Model

The Doge model employs dynamic masked attention mechanisms for sequence transformation, with the option to use either multi-layer perceptrons or cross-domain mixture of experts for state transitions.

Doge 320M Instruct is a lightweight language model based on dynamic masked attention, trained with supervised fine-tuning (SFT) and direct preference optimization (DPO), suitable for question-answering and dialogue tasks.

Bonsai is a small ternary-weighted language model with 500 million parameters, built on the Llama architecture and using the Mistral tokenizer, trained on fewer than 5 billion tokens.

Doge 160M is a small language model that employs dynamic masked attention mechanisms, trained by the SmallDoge community, and supports text generation tasks.

Doge 20M is a small language model based on dynamic masked attention mechanism, supporting instruction following and Q&A tasks.

Simple Stories is a series of small text generation models trained on the TinyStories dataset, focusing on generating children's stories.

This model is a fine-tuned version of Microsoft's MiniLM-L12-H384-uncased on the IMDB dataset, primarily used for sentiment analysis tasks.